Dataproc Cookbook (for . .) by Narasimha Sadineni and Anuyogam Venkataraman

Author:Narasimha Sadineni and Anuyogam Venkataraman
Language: eng
Format: epub
Publisher: O'Reilly Media, Inc.
Published: 2024-03-06T00:00:00+00:00

Solution

Export the existing cluster configuration to a file

gcloud dataproc clusters export <source-cluster-name> --destination prod-cluster-config.yaml

Create a new cluster using the YAML configuration file

gcloud dataproc clusters import <target-cluster-name>\ --source prod-cluster-config.yaml\ --region=region

Discussion

When working with existing clusters, you may want to view cluster details such as worker details, labels, custom configurations, and component gateway URLs.

Gcloud command for viewing existing cluster configuration

gcloud dataproc clusters describe <cluster-name-here> --region <region>

Creating a new cluster with the same configuration as the existing cluster is a two step process. First we have to export the existing cluster configuration to a file. Dataproc offers the gcloud command option to export the configuration in YAML file format. At the time of this writing this option of configuration export is only supported from gcloud and can’t be done from Web UI.

Run a command to export the configuration

gcloud dataproc clusters export prod-cluster --destination prod-cluster-config.yaml

Upon successful execution of the command, the cluster configuration will be stored in a file named prod-cluster-config.yaml. The cluster name and region are not included in the export because the name must be unique. When creating a new cluster using this configuration, the cluster name and region must be provided.

Run a command to create a new cluster using the configuration present in YAML file

gcloud dataproc clusters import prod-cluster-duplicate\ --source prod-cluster-config.yaml\ --region=region

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.